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ABSTRACT 



The purpose of this paper is to help higher education 
institutions determine how well their institutional effectiveness models 
match the North Central Association of Colleges and Schools Commission on 
Institutions of Higher Education Criterion Four requirements. The paper 
describes each of the major requirements as a domain in a matrix with 
cross-related subcategories. The matrix is built around six issues: (1) 

activities to be measured need to be established, (2) perspectives of the 
constituencies and related criteria need to be identified, (3) the level of 
analysis should be dependent on constituent perspectives and expectations, 

(4) the time frame determines how data are measured, (5) the types of data 
needed are related to perspectives and levels of analysis, and (6) the 
referent used to assess indicators is determined by the dominant 
constituency. The paper suggests that institutions use this matrix to assess 
the degree to which their institutions match the elements required in 
Criterion Four, and also to identify strengths and areas for improvement. 
(Contains 30 references.) (CH) 
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Abstract 

Considering current federal, regional, and state policies, it is important for 
institutions under the 19-state purview of the North Central Association of Colleges and 
Schools, Commission on Institutions of Higher Education (NCA) to understand more 
clearly the NCA’s requirements for demonstrating assessment of “institutional 
effectiveness” as stated in Criterion Four. The purpose of this paper is to help 
institutions of higher education determine how well their institutional effectiveness 
models match the NCA’s requirements that are found in its Criterion Four to 
demonstrate effectiveness. Using the six issues of Cameron (1980) as a conceptual 
framework for assessing institutional effectiveness, the Matrix of Domains is derived 
from analyzing and categorizing the elements of Criterion Four and related General 
Institutional Requirements (GIRs). The paper describes each of the major 
requirements as a domain of the matrix with sub-categories, which are cross-related in 
some areas. The existence of gaps or varying degrees of matching between NCA’s 
domains of activity and an institution’s has implications for an institution completing its 
self-study process. An institution may need to determine whether certain domains of 
activity rank differently in importance with NCA or the existence of varying degrees of 
match in the elements are acceptable. 

ERIC 
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Introduction and Perspectives 

Assessing institutional effectiveness has become a major issue in higher and 
postsecondary education. Regulatory entities at the federal, regional, and state levels 
have formulated policies and procedures for assessing "institutional effectiveness" and 
"student outcomes." Many of these entities have described "institutional effectiveness" 
as assessing the effectiveness of institutions in terms of achievement of their missions 
and goals (Commission on Higher Education, Middle States Association of Colleges 
and Schools [MSA], 1996; Council on Postsecondary Accreditation [COPA], 1992; 
Ewell, 1992; Ewell & Boyer, 1988; McLeod & Atwell, 1992; North Central Association of 
Colleges and Schools, Commission on Institutions of Higher Education [NCA], 1991, 
1997; U.S. Department of Education [USDE], 1994; Wolff, 1992). It is no longer a 
question of “ Why do we assess effectiveness,” but, “ How do we assess effectiveness?” 

On the regional level, all accrediting agencies have incorporated institutional 
effectiveness into their criteria for evaluating their institutions (Hudgins, 1993; Losak, 
1987). For example, the 19-state North Central Association (NCA) established five 
criteria upon which to evaluate member institutions for accreditation (purposes, 
organization, accomplishment of educational purposes, strengthening educational 
effectiveness, and integrity) (NCA, 1997). Having moved beyond the emphasis of 
"inputs," (number of volumes in the library, number of faculty and their degrees, etc.), 
all six regional accrediting agencies require institutions to assess the processes 
involved in instruction and student support (Hogan, 1992; Simmons, 1993). They have 
identified the other components of effectiveness models, including outputs and 
outcomes (Nettles, 1987). 
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These agencies have also included student outcomes assessment as a part of 
assessing outputs in evaluating overall institutional effectiveness (MSA, 1996; NCA, 
1991; 1997). The Commission on Higher Education, Middle States Association of 
Colleges and Schools (MSA) further expects to see an institutional research structure 
and data base to support these assessment functions (MSA, 1996). North Central 
Association requires its institutions to assess overall institutional effectiveness (Criterion 
Four) and student academic achievement (Criterion Three). In recent publications, 

NCA indicates a stronger emphasis on assessing student academic achievement by 
requiring this assessment plan and program to function as an independent component, 
yet linked to the overall model for assessing institutional effectiveness (Lopez, 1996; 
NCA, 1996, 1997). 

Purpose 

The purpose of this paper is to examine the elements required in NCA’s Criterion 
Four and provide a Matrix of Domains (and related areas of activities) in order to help 
the institution determine the degree to which it matches these elements. Comparing 
the Matrix of Domains to an institution’s institutional effectiveness model should help 
determine whether there are any significant information/data gaps that, among other 
factors, might prevent the institution from meeting NCA’s requirements. 

In Criterion Four the North Central Association requires assessment of 
institutional effectiveness, “The institution can continue to accomplish its purposes and 
strengthen its educational effectiveness” (NCA, 1997, p. 52). This criterion comprises 
NCA definitions of phrases, such as “educational effectiveness,” the role of institutional 
planning in meeting the criterion, the expectation that other criteria have been 
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previously met, and establishing patterns of evidence, including the establishment of 
“structured assessment processes" (NCA, 1997, p. 53). In addition to these elements 
in Criterion Four, the first domain of the Matrix of Domains includes five categories 
(Mission, Governance, Faculty, Educational Program, and Finances) involving 6 of the 
24 General Institutional Requirements (GIRs) that directly relate to the previously stated 
elements of Criterion Four: (1) Mission Statement, (2) Executive Officer Providing 
Leadership, (3) Significant Role of Faculty in Developing and Evaluating All Educational 
Programs, (4) Degree Programs Compatible with Institution’s Mission, (5) Institution’s 
Appropriate Use of Resources to Support Programs, and (6) Practices that 
Demonstrate “Fiscal Viability” (NCA, 1997). 

Conceptual Framework 

There are overlapping issues and questions among the different frameworks for 
assessing institutional effectiveness (Cameron, 1978, 1980; Cameron & Smart, 1998; 
Goodman & Pennings, 1980; R. A. Scott, 1984; W. R. Scott, 1977; Seashore, 1983). 
Goodman and Pennings (1977) developed six issues as a means of assessing the 
effectiveness of an institution. They and others (Cameron, 1978; Etzioni, 1964; Nadler 
& Tushman, 1980) have also identified organizational models established to assess 
institutional effectiveness. Cameron (1980) cogently addresses and consolidates these 
issues. Cameron’s six issues that must be addressed in assessing effectiveness are 
the following: (1) The domain of activity to be measured needs to be established. (2) 
The constituencies’ perspectives and related criteria need to be identified. (3) The level 
of analysis to be used is dependent on constituent perspectives and expectations. (4) 
The time frame involved determines how data is measured. (5) The types of data 
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needed are related to perspectives and levels of analysis. (6) The referent to be used 
by which to assess indicators is determined by the dominant constituency. These six 
issues establish the framework from which to analyze North Central Association’s 
Criterion Four and related six GIRs and to construct the Matrix of Domains and areas of 
activities. 

Approach 

This paper applies these six issues of Cameron (1980) to the elements found in 
Criterion Four and the General Institutional Requirements GIRs and demonstrates that 
the Matrix incorporates the Domains of activities (issue 1) resulting from the perspective 
and criteria (issue 2) of NCA found in Criterion Four and the related GIRs. From these 
domains and perspectives and criteria, institutions should use the organizational level of 
analysis (issue 3) in a static and/or longitudinal time frame (issue 4) to compare the 
types of data (issue 5) required by NCA as the normative referent (issue 6). 

(1) The domain of activity to be measured needs to be established . In this 
context, domains are similar to missions (Cameron & Whetten, 1983). They need to be 
defined with a set of criteria by which to be assessed (Goodman & Pennings, 1977). 

The Domains in this Matrix have been defined by the guidelines suggested by NCA. 

The missions stated in the colleges’ institutional effectiveness models need to be 
compared to NCA guidelines related to missions. 

(2) The constituencies’ perspectives and related criteria need to be identified . 
Identifying the constituency clarifies the values used to assess the criteria (Goodman & 
Pennings, 1977). The external constituency (in this case, NCA) and the internal 
constituency (the institution) share the same criteria to be used in the self-evaluation 
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process. This Matrix of Domains incorporates the suggested guidelines in Criterion 
Four and the related General Institutional Requirements (GIRs) of NCA as a basis from 
which each institution can assess the elements of its institutional effectiveness model. 

(3) The level of analysis to be used is dependent on constituent perspectives 
and expectations . Assessment of institutional effectiveness can and does take place at 
different levels (organizational, sub-unit, etc.). The Matrix of Domains uses the 
organizational level of analysis as it relates to NCA Criterion Four, “The institution can 
continue to accomplish its purposes and strengthen its educational effectiveness” 

(NCA, 1997, p.52). 

(4) The time frame involved determines how data are measured . Some 
researchers have used the dynamic, or long-term (longitudinal), approach (Cameron, 
1978). Most, however, use the static, or short-term approach to describe inputs, 
processes, and outcomes (Cameron, 1978). The Matrix of Domains views the static 
time frame as a point in time of the NCA self-study. However, certain longitudinal data 
can be reported. 

(5) The types of data needed are related to perspectives and levels of analysis . 
Researchers have disagreed over whether to use only objective indicators, such as 
documents containing quantitative data (Mirvis, 1980), or perceptual indicators, such as 
surveys and focus group and individual interviews, by which to assess effectiveness 
(see Pennings, 1975, 1976; also Steers, 1977 as cited in Goodman, Pennings & 
associates, 1977). Others have indicated having used both (Cameron, 1978; Cameron 
& Smart, 1998). The elements in the Matrix of Domains indicate NCA’s emphasis of 
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objective indicators, yet some perceptual indicators may help to demonstrate 
institutional effectiveness in each institution’s model. 

(6) The referent to be used by which to assess indicators is determined by the 
dominant constituency . As Cameron (1978) points out, each constituency’s viewpoint 
affects the criteria used. He further states that referents can be comparative (between 
institutions), normative (against a standard), goal-centered (compared against a stated 
goal), or improvement evaluation (compared against past performance). Since the 
accrediting agency established Criterion Four and its indicators, the Matrix of Domains 
establishes the dominant referent as normative. However, since the Matrix of Domains 
includes NCA’s Criterion One and GIR One (the institution’s mission and purpose), an 
aspect of the referent is also goal-centered. 

Analysis and Results 

Using the six issues of Cameron (1980) as a framework, the Matrix of Domains 
resulted from analyzing the guideline statements in NCA’s Criterion Four and six of the 
24 General Institutional Requirements (GIRs). The NCA documents included the 
Handbook of Accreditation. 1994-1996 (NCA, 1994), which contained Criterion Four 
and the 24 GIRs. Another document was the “Working Draft: Revised Sections of the 
Handbook of Accreditation : Criteria Three and Four” (NCA, 1996), which was approved 
by NCA at its 1996 annual meeting. The most recent document was Handbook of 
Accreditation. Second Edition. 1997 . For the purposes of creating this current Matrix, 
these documents stand as the most recent ones containing Criterion Four. The Matrix 
will need to undergo modifications whenever NCA revises any related aspects of 
Criterion Four, Criterion Three, Criterion One, and/or the GIRs. 



Institutional Effectiveness Matrix 

9 



Using the perspectives and criteria of the normative constituent (NCA), the 
author created descriptors for the major domains as indicated by subject heading in 
Criterion Four (Cameron, 1980). The results of applying these descriptors to areas in 
the matrix led the author to identify five major categories involving six GIRs as directly 
applicable to the other domains derived from Criterion Four. When completed, the 
author removed the GIRs that were not directly applicable to Criterion Four and re- 
configured the matrix. This revised matrix now included descriptors created by domains 
of activities in Criterion Four and a domain created by those GIRs that directly applied 
to the other domains (see Tablel). 

Analysis of the first domain, the five major categories that included six specific 
GIRs, showed that NCA requires the institution to have a clearly stated mission and an 
executive officer providing leadership for the institution in attempting to achieve its 
mission. NCA also requires the faculty to play a major role in the development and 
evaluation of all educational programs and that these degree programs directly 
associate with the institution’s mission. The final two GIRs related to this domain 
require the institution to integrate budget allocations to effectively support the 
educational programs that accomplish the stated mission of the institution. Thus, the 
first domain is “NCA General Institutional Requirements Related to This Criterion.” 
Analysis of the domains of Criterion Four found that, from NCA’s perspective, one 
domain involves the institution having met the requirements of Criterion One (“The 
institution has clear and publicly stated purposes consistent with its mission and 
appropriate to an institution of higher education.”) and Criterion Three (“The institution 
is accomplishing its educational and other purposes.”) as a pre-requisite of meeting the 




11 



Institutional Effectiveness Matrix 

10 



Tablel. Matrix of Domains Created From NCA’s Criterion Four and Related GIRs 



NCA General 
Institutional 
Requirements 

(GIRs) 


Other Required 
NCA Criteria 
Supporting 
This Criterion 


Activities 
Required by 
NCA to Support 
Educational 
Effectiveness 


Pattern of Evidence 
Supporting This Criterion 


Mission 

1 .Mission statement 


Criterion One to 
be met 


1 . Planning for 

educational 

improvement 


Planning process/ Plans as well 
as ongoing effective planning 
processes 


Governance 
7. Executive officer 
provides leadership 
for institution 




2. Building on 
Assessment 


Structured assessment 
processes 


Facultv 

1 1 . Significant role 
in dev. and eval. of 
all educ. programs 


Criterion Three 
to be met 






Educational 

Program 

14. Decree 

programs 

compatible with 

institution’s 

mission 








Finances 
20. Demonstrate 
appropriate use of 
resources to 
support programs 




3. Growing from 
the self-studv 
process 
(corresponding 
allocation of 
resources) 


Current resource base that 
positions the institution for the 
future/ Resources organized and 
allocated to support its plans 


21. Practices, 
records & reports 
demonstrate fiscal 
viability 









requirements of Criterion Four (NCA, 1997, p. 52). Meeting these two criteria directly 
related to a domain of activity derived from part of the criterion statement, “Institution 
can continue to accomplish its purpose.” Consequently, the second domain to be 
established is “Other Required NCA Criteria Supporting This Criterion.” 




12 



Institutional Effectiveness Matrix 

11 



The third domain of activity in the Matrix that is created comes from the 
remaining part of the criterion statement, “Institution can strengthen its educational 
effectiveness.” This domain includes three areas for activities: (1 ) “Planning for 
educational improvement,” (2) “Building on assessment,” and (3) “Growing from the 
self-study process” (NCA, 1997, p. 53). The key factor that NCA requires in the first 
area is an effective planning process, which includes involving representatives from all 
constituencies, scanning internal and external factors affecting the institution, updating 
an annual document used by the institution, and presenting accomplishments and 
obstacles/changes reached through adjusting prior annual planning (NCA, 1996). 

North Central Association requires the following key factors in the second and third 
areas of activities, “Building on assessment," and “Growing from the self-study”: 

•the governing board supports assessment 

•senior executive officers provide leadership and support assessment 
•sufficient resources are allocated to sustain ongoing assessment efforts 
•funds are available to support changes that need to be made to enhance 
student academic achievement 

•all planning and budgeting processes include ways in which assessment 
information can influence institutional priorities 
(NCA, 1997, p. 53). 

Consequently, the third domain to be used is “Activities Required by NCA to Support 
Educational Effectiveness.” 

The final domain of activity, “Patterns of Evidence Supporting This Criterion,” 
involves the breadth of review that NCA requires an institution to include for successful 
completion of Criterion Four (NCA, 1997, p. 55). These patterns include, but are not 
restricted to, the following: (1) a fiscal, physical, and human resource base from which 
to position the college for the future, (2) tested decision-making processes to respond 
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to external and internal changes, (3) assessment processes that include different 
internal constituencies and report useful information to those constituencies and the 
planning process itself, (4) ongoing planning process, and (5) resource allocation 
focused to support the institution and programs (NCA, 1997). 

Keeping in mind the constituencies’ perspectives (issue 2) and levels of analyses 
(issue 3) of Cameron (1980), the institutions need to demonstrate a continuous 
assessment cycle that involves various internal constituencies, especially faculty, and 
includes feedback of useful information to a variety of internal and external 
constituencies: faculty, administration, students, governing board, and community. 

NCA requires indicators related to reporting the results of the assessment process in 
institutions’ models to demonstrate (1) leadership stability at all levels (board, faculty, 
administration), (2) human, physical, and fiscal stability, (3) responsive governance 
structure and processes, and (4) continuous institutional assessment and planning 
activities (NCA, 1997). 

Conclusions and Implications 

Institutions should be able to use this matrix to assess the degree to which their 
institutions match the elements required in Criterion Four. The results of their 
assessment could be useful in completing their self-study. 

Early analysis of the degree to which each element matches North Central 
Association’s domains should allow institutions the time to clarify and demonstrate their 



activities in relation tc these domains. The institution can also identify its strengths and 



areas of improvement. If serious gaps or low degrees of match occur, an institution 
may need to determine whether certain domains of activity rank differently in 
importance with NCA or the existence of varying degrees of match in the elements are 
acceptable, with the understanding that these elements are future areas for 
improvement. 
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By extrapolating the use of Cameron’s six issues and the methodology for 
creating this matrix, institutions should be able to create similar matrices for most, if not 
all, of the remaining criteria needed to be completed in their self-study. 



Institutional Effectiveness Matrix 

14 



References 

Cameron, K. S. (1978). Measuring organizational effectiveness in institutions of 
higher education. Administrative Science Quarterly, 23: 604-632. 

Cameron, K. S. (1980). Critical questions in assessing organizational 
effectiveness. Organizational Dynamics. 9 (2), 66-80. 

Cameron, K. S., & Smart, J. (1998). Maintaining effectiveness amid downsizing 
and decline in institutions of higher education. Research in Higher Education, 39 (1), 
65-86. 

Cameron, K. S., & Whetton, D. A. (1983). Models of the organizational life 
cycle: Applications to higher education. The Review of Higher Education, 6, 269-299. 

Commission on Higher Education, Middle States Association of Colleges and 
Schools. (1996, March Draft). Framework for outcomes assessment. Philadelphia: 
Author. 

Council on Postsecondary Accreditation. (1992, January). Accreditation. 
assessment, and institutional effectiveness: Resource papers for the COPA task force 
on institutional effectiveness. Washington, DC: Author. 

Etzioni, A. (1964). Modern organizations. Englewood Cliffs, NJ: Prentice-Hall. 

Ewell, P. T. (1992, January). Outcomes assessment, institutional effectiveness, 
and accreditation: A conceptual exploration. In Council on Postsecondary 
Accreditation, Accreditation, assessment, and institutional effectiveness: Resource 
papers for the COPA task force on institutional effectiveness. Washington, DC: 

Author. (ERIC Document Reproduction Service No. ED 343 513) 




lb 



Institutional Effectiveness Matrix 

15 

Ewell, P. T., & Boyer, C. (1988, July/August). Acting out state-mandated 
assessment: Evidence from five states. Change, 20, 40-47. 

Goodman, P. S., & Pennings, J. M. (1977). Perspectives and issues: An 
introduction. In P. S. Goodman, J. M. Pennings, & associates (Eds.), New perspectives 
on organizational effectiveness (pp. 1-12). San Francisco: Jossey-Bass. 

Goodman, P. S., Pennings, J. M., & associates (Eds.). (1977). New 
perspectives on organizational effectiveness. San Francisco: Jossey-Bass. 

Goodman, P. S., & Pennings, J. M. (1980). Critical issues in assessing 
organizational effectiveness. In E. E. Lawler III, D. A. Nadler, & C. Cammann (Eds.), 
Organizational assessment: Perspectives on the measurement of organizational 
behavior and the Quality of work life (pp. 185-215). New York: Wiley. 

Hogan, T. P. (1992, January). Methods for outcomes assessment related to 
institutional accreditation. In Council on Postsecondary Accreditation, Accreditation, 
assessment, and institutional effectiveness: Resource papers for the COPA task force 
on institutional effectiveness. Washington, DC: Author. 

Hudgins, J. L. (1993, April/May). Institutional effectiveness: A strategy for 
renewal. Community College Journal. 63 (5), 41-44. 

Lopez, C. (1996, March). Opportunities for improvement: Advice from 
consuitant-evaluators on programs to assess student learning. Chicago: North Central 
Association Commission on Institutions of Higher Education. 

Losak, J. (1987, Fall). Assessment and improvement in education. In D. Bray & 
M. J. Belcher (Eds.), Issues in student assessment. New Directions for Community 
Colleges (pp. 25-9), no. 59. San Francisco: Jossey-Bass. 




17 



Institutional Effectiveness Matrix 

16 

McLeod, M. W., & Atwell, C. (1992, Fall). A mini-dictionary on institutional 
effectiveness terms. Community College Review, 20 (2), 30-8. 

Mirvis, P. H. (1980). Assessing physical evidence, documents, and records in 
organizations. In E. E. Lawler III, D. A. Nadler, & C. Cammann (Eds.), Organizational 
assessment: Perspectives on the measurement of organizational behavior and the 
Quality of work life (pp. 418-443). New York: Wiley. 

Nadler, D. A., & Tushman, M. L. (1980). A congruence model for organizational 
assessment. In E. E. Lawler III, D. A. Nadler, & C. Cammann (Eds.), Organizational 
assessment: Perspectives on the measurement of organizational behavior and the 
Quality of work life (pp. 261-278). New York: Wiley. 

Nettles, M. (1987). The emergence of college outcome assessments: 

Prospects for enhancing state colleges and universities (Working paper). Trenton, NJ: 
New Jersey State College Governing Boards Association, Inc. 

North Central Association of Colleges and Schools, Commission on Institutions 
of Higher Education. (1991 , September 9). Ten characteristics of an assessment 
program [A memorandum to institutions scheduled for comprehensive visits after 1995- 
96]. Chicago: Author. 

North Central Association of Colleges and Schools, Commission on Institutions 
of Higher Education. (1994). Handbook of accreditation 1994-1996. Chicago: Author. 

North Central Association of Colleges and Schools, Commission on Institutions 
of Higher Education. (1996, March). Revised sections of the Handbook of 
accreditation: Criteria three and four. Working Draft Chicago: Author. 




18 



Institutional Effectiveness Matrix 

17 

North Central Association of Colleges and Schools, Commission on Institutions 
of Higher Education. (1994). Handbook of accreditation, second edition. 1997. 

Chicago: Author. 

Scott, R. A. (1984, March). Editor’s notes. In R. A. Scott (Ed.), Determining the 
effectiveness of campus services. New Directions for Institutional Research (pp. 1-7) 
no. 41 . San Francisco: Jossey-Bass. 

Scott, W. R. (1977). Effectiveness of organizational effectiveness studies. In P. 
S. Goodman, J. M. Pennings, & associates (Eds.), New perspectives on organizational 
effectiveness (pp. 63-95). San Francisco: Jossey-Bass. 

Seashore, S. E. (1983). A framework for an integrated model of organizational 
effectiveness. In K. Cameron & D. Whetten (Eds.). Organizational effectiveness: A 
comparison of multiple models (pp.55-70). New York: Academic Press. 

Simmons, H. L. (1993, Fall). Accreditation and the community college: 
Challenges and opportunities. In C. Prager (Ed.), Accreditation of the two-year college. 
New Directions for Community Colleges (pp. 83-91) no. 83. San Francisco: Jossey- 
Bass. 

U.S. Department of Education. (1994, January 24). State postsecondary review 
program; Proposed rule, Federal Register (34 CFR Part 667, pp. 3604-3623). 
Washington, DC: Author. 

Wolff, R. A. (1992, January). Incorporating assessment into the practice of 
accreditation: A preliminary report. In Council on Postsecondary Accreditation, 
Accreditation, assessment, and institutional effectiveness: Resource papers for the 




13 



Institutional Effectiveness Matrix 

18 



COPA task force on institutional effectiveness . Washington, DC: Author. (ERIC 
Document Reproduction Service No. ED 343 513) 




U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASTS 




This document is covered by a signed “Reproduction Release 
(Blanket)” form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket”). 



